What Would We Like IR Metrics to Measure?

The field of Information Retrieval has a long-standing tradition of rigorous evaluation, and an expectation that proposals for new mechanisms and techniques will either be evaluated in batch-mode experiments against realistic test collections, with results reported derived from standard tools; or will be evaluated through the use of user studies. This emphasis on evidence, and the desire for verification of proposals, has meant that IR effectiveness measurement is an important area studied in its own right. The result has been the development of a complex suite of relevance metrics, each of them with seemingly different behavior. Well-known examples include Precision, Recall, Average Precision, Normalized Discounted Cumulative Gain [3], BPref [2], the Q-Measure [6], Rank-Biased Precision (RBP) [4], and so on. In this presentation the underlying question of what it is that a metric should measure is returned to, with a set of desiderata for usefulness used as a starting point for examining the existing palette of metrics. Recent work that has described a goal-sensitive adaptive metric called INST [1, 5] will then be presented. Acknowledgment. Much of the work presented in this talk has been carried out and published in collaboration with Peter Bailey (Microsoft), Falk Scholer (RMIT University), and Paul Thomas (CSIRO, Microsoft). University of Melbourne collaborators in earlier work in the area of evaluation have included Justin Zobel and William Webber.